Systems and methods for machine learning enhanced by human measurements

ABSTRACT

In various embodiments, training objects are classified by human annotators, psychometric data characterizing the annotation of the training objects is acquired, a human-weighted loss function based at least in part on the classification data and the psychometric data is computationally derived, and one or more features of a query object are computationally classified based at least in part on the human-weighted loss function.

RELATED APPLICATION

This application claims the benefit of and priority to U.S. ProvisionalPatent Application No. 61/840,871, filed Jun. 28, 2013, the entiredisclosure of which is hereby incorporated herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with government support under Grant NumberIIS-0963668 awarded by the National Science Foundation. The governmenthas certain rights in the invention.

TECHNICAL FIELD

In various embodiments, the present invention relates to machinelearning, in particular to machine learning augmented by measurementsacquired from human populations.

BACKGROUND

Traditional machine-learning algorithms typically incorporate humanknowledge (at least implicitly) as an input—for instance, amachine-learning system trained to decide whether a face is present inan image will be given set of human-generated “ground truth” labels thatsignal whether a human judged the image to contain a face or not. Inthis case, the job of the machine-learning system is to emit analogouslabels (for example, “face present” and “face not present”) in responseto new, previously unseen images.

While great progress has been made in the field of machine learning, theperformance of machine-learning systems often falls far short humanlevels of performance, particularly in the domain of machine vision. Akey limiting factor is the unavailability of labeled data; it isdifficult to provide an algorithm with enough labeled training data toachieve optimal performance. Without sufficiently large datasets,machine-learning algorithms tend to “overfit” the data, adapting tospurious structure present in the training set that is notrepresentative of the larger distribution of all examples in the realworld. Machine-learning systems typically combat the effects ofoverfitting by a process called “regularization,” in which penalties areplaced on solutions that are thought to be more likely to be the resultof overfitting, typically because they are more complex or because theyexhibit less stable behavior under injected noise.

Thus, there is a need for machine-learning systems and techniques thatincorporate not only larger sets of human-labeled data, but also utilizesuch data to regularize solutions to machine-learning problems in novelways to better mimic human performance.

SUMMARY

In accordance with various embodiments of the present invention, largesets of human-derived data containing different types of information areutilized to provide ground-truth labels for regularization ofmachine-learning systems. An advantageous aspect of embodiments of theinvention is that details and patterns of human performance underdifficult conditions may be used to regularize solutions tomachine-learning problems. For example, information about how difficultit is for a human to label each training example in a dataset may beutilized as at least a portion of the basis for a powerful regularizer.Specifically, learned solutions that deviate substantially from thehuman patterns of difficulty may be penalized. For example, greaterpenalties may be introduced for misclassification, on the part of themachine-learning system, of examples that are easily classified byhumans. Similarly, lesser penalties may be introduced formisclassification of examples that are relatively more difficult forhumans to classify. By imposing such additional weightings in theobjective function of the machine-learning algorithm, the system may beinfluenced to produce more “human-like” solutions, effectivelyleveraging fine-grained information resulting from the extremelysophisticated learning mechanisms present in humans, along with the vastwealth of experience (i.e., training data) that the human has received.Techniques for incorporating a wide range of measurements of humanknowledge and perception into machine-learning algorithms are alsoreferred to herein as “deep annotation.”

A variety of well-known techniques exist for measuring humanperformance, including reaction time measurement (i.e., how quickly isthe label generated), the generation of psychophysical curves understimulus degradation (e.g., how does performance fall-off as the imagebecomes less clear), item-response curves across large populations ofhumans (e.g., how consistent are judgment across a population), and manyother techniques falling under the umbrella of the field ofpsychophysics. Any or all of these techniques may be used as additionalstreams of input in accordance with embodiments of the presentinvention.

In an aspect, embodiments of the invention feature acomputer-implemented method for data classification and identification.Data corresponding to a plurality of training objects is provided, overa computer network, to a plurality of training devices each associatedwith one of a plurality of human annotators. Each of the trainingobjects includes or consists essentially of features for classification.The training objects are displayed on a display of each of the trainingdevices. Classification data is received via communication interfaces ofat least some of the training devices. The classification data includesor consists essentially of at least some of the training objectsannotated, via annotation interfaces of the training devices, by atleast some of the annotators with classifications for features thereof.Psychometric data characterizing the annotation of the training objectsby the annotators is acquired. A human-weighted loss function based atleast in part on the classification data and the psychometric data iscomputationally derived. The human-weighted loss function includespenalties for misclassification, magnitudes of the penalties increasingwith increasing deviation from the classification data. Datacorresponding to a query object different from the plurality of trainingobjects is received by a classification device. Thereafter, at least onefeature of the query object is computationally classified by a computerprocessor based at least in part on the human-weighted loss function.

Embodiments of the invention feature one or more of the following in anyof a variety of combinations. The classification of the at least onefeature of the query object may be displayed. The magnitudes of thepenalties for misclassification may be based at least in part on thepsychometric data. Computationally deriving the human-weighted lossfunction may include or consist essentially of (i) computationallyclassifying, by the computer processor, at least some of the trainingobjects based at least in part on an initial loss function, therebygenerating training data, (ii) comparing the training data to theclassification data to identify, within the training data, featuresmisclassified in comparison to the classification data, (iii) assigningthe penalties for misclassification to the misclassified features in thetraining data, and (iv) incorporating the penalties formisclassification within the initial loss function to generate thehuman-weighted loss function. The penalties for misclassification may beassigned based at least in part on the psychometric data. The initialloss function may include or consist essentially of a hinge lossfunction (e.g., a linear hinge loss function). The psychometric data mayinclude or consist essentially of (i) response time for classifying oneor more features, (ii) accuracy of feature classification, and/or (iii)presentation time of one or more training objects. Each of the trainingobjects may include or consist essentially of a digital image, and oneor more features for classification may include or consist essentiallyof human faces. At least one of the training devices may include orconsist essentially of a computer or mobile computing device. The queryobject may include or consist essentially of a digital image, and atleast one said feature of the query object may include or consistessentially of a human face.

In another aspect, embodiments of the invention feature a system fordata classification and identification. The system includes or consistsessentially of a database of training objects, a computer processor, aclassification device, a penalization module, and a classificationmodule. The database includes or consists essentially of a storagemedium populated with stored computer records specifying, for each of aplurality of training objects, (i) classification data comprisingannotations received from a plurality of human annotators, and (ii)psychometric data characterizing the annotation of the training objectby the plurality of human annotators. The classification device isconfigured to receive query objects, which may be different from thetraining objects in the database. The penalization module is executableby the computer processor and is configured to derive a human-weightedloss function based at in part on the classification data in thedatabase and the psychometric data of at least some of the trainingobjects in the database. The loss function includes penalties formisclassification, and the magnitudes of the penalties increase withincreasing deviation from the classification data. The classificationmodule is executable by the computer processor and is configured toclassify features of query objects based at least in part on thehuman-weighted loss function.

Embodiments of the invention feature one or more of the following in anyof a variety of combinations. The system may include a display modulethat is executable by the computer processor and configured to displaytraining objects to the plurality of human annotators. The system mayinclude a plurality of training devices, each associated with a humanannotator, for displaying training objects. Each training device mayinclude a communication interface for receiving training objects andtransmitting classification data. At least one of the training devicesmay include or consist essentially of a computer or mobile computingdevice. The psychometric data may include or consist essentially of (i)response time for classifying one or more features, (ii) accuracy offeature classification, and/or (iii) presentation time of one or moretraining objects. Each of the training objects may include or consistessentially of a digital image, and one or more features forclassification may include or consist essentially of human faces. Atleast one query object may include or consist essentially of a digitalimage, and at least one feature of the query object may include orconsist essentially of a human face.

These and other objects, along with advantages and features of thepresent invention herein disclosed, will become more apparent throughreference to the following description, the accompanying drawings, andthe claims. Furthermore, it is to be understood that the features of thevarious embodiments described herein are not mutually exclusive and mayexist in various combinations and permutations. As used herein, theterms “approximately” and “substantially” mean±10%, and in someembodiments, ±5%. The term “consists essentially of” means excludingother materials that contribute to function, unless otherwise definedherein.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, like reference characters generally refer to the sameparts throughout the different views. Also, the drawings are notnecessarily to scale, emphasis instead generally being placed uponillustrating the principles of the invention. In the followingdescription, various embodiments of the present invention are describedwith reference to the following drawings, in which:

FIGS. 1A and 1B are schematic block diagrams of classification systemsin accordance with various embodiments of the invention;

FIG. 2 is a graph of three item response functions for images presentedon classification systems in accordance with various embodiments of theinvention;

FIG. 3 is a pictorial flowchart of a technique of acquiring psychometricdata in accordance with various embodiments of the invention;

FIG. 4A is a pictorial flowchart of a conventional supervised learningtechnique;

FIG. 4B is a pictorial flowchart of a deep annotation machine learningtechnique in accordance with embodiments of the invention;

FIG. 5 is a pictorial flowchart of a face-detection method in accordancewith various embodiments of the invention;

FIG. 6A depicts images of faces at various levels of image coherence, inaccordance with various embodiments of the invention;

FIG. 6B is a graph of the accuracy of various techniques of facialrecognition as a function of coherence, in accordance with variousembodiments of the invention;

FIG. 7A depicts images of faces obscured by various techniques, inaccordance with various embodiments of the invention;

FIG. 7B is a graph of the accuracy of various techniques of facialrecognition as a function of visible facial area, in accordance withvarious embodiments of the invention;

FIGS. 8A and 8B are pictures of various training objects, and trainingresults, provided to annotators in accordance with various embodimentsof the invention;

FIG. 9 is a graph comparing the accuracy of face classificationutilizing a human-weighted loss function, in accordance with variousembodiments of the invention, with face classification utilizing aconventional hinge loss function;

FIG. 10 is a graph comparing the accuracy of face classificationutilizing an embodiment of the invention with face classificationutilizing a variety of conventional techniques;

FIG. 11 is a graph comparing the accuracy of face classificationutilizing a human-weighted loss function, in accordance with variousembodiments of the invention, with face classification utilizing aconventional hinge loss function; and

FIG. 12 depicts a comparison of the classification of a series of imagesvia an embodiment of the present invention with the classification ofthe series of images via a conventional Viola-Jones technique.

DETAILED DESCRIPTION

In various embodiments of the invention, patterns of human errors underpurposely challenging conditions are used to weight the margins of alinear kernel machine, which may then be utilized for practicalmachine-learning tasks such as machine vision (e.g., face detection).For example, large-scale visual psychophysics experiments may beutilized to inform the learning at an algorithmic level, which is quitedifferent from the traditional “query the oracle” model forimage-specific labels. Embodiments of the invention build moreintelligent weighed loss functions from overall models of humandecisions.

Many computer vision systems attempt to mimic natural human visualabilities in machines—object and face recognition, scene understanding,image segmentation, and medical image analysis, to name just a few.Indeed, any setting where human-provided labels are used as groundtruth—whether the system aspires to be biologically-inspired or not—isultimately driven by the human visual system and its ability to provideaccurate labels. However, while human judgments already provideessential raw material for machine learning, human-generated labelsrepresent only a small fraction of the potential information that may beextracted from human subjects. Human behavior is intrinsicallymultidimensional—a given judgment may be correct or incorrect, but it isalso associated with a wealth of other information, such as a reactiontime. Moreover, if stimuli are degraded (e.g., by additive noise) orappear quickly, a rich landscape of performance emerges even forbehaviors that humans normally perform almost perfectly. Patterns oferrors across stimuli may yield important information about thecomputations being performed, and the distribution of performance acrossa large population of humans may yield important insights into whichelements of a perceptual ability are consistently expressed and whichare not. The science of visual psychophysics has long exploited theseexternal clues to provide key insights into the inner workings of humanperception. Embodiments of invention utilize these sophisticated toolswith state-of-the-art machine vision and machine learning.

A key challenge in machine vision, and machine learning in general, isthat the quantity of available training data is typically small, andthis limits a system's ability to effectively learn. Embodiments of theinvention leverage human psychophysics to address this problem via aconcerted effort in several ways. First, advanced crowd-sourcingtechnologies are utilized to dramatically improve the quality, quantity,and depth of annotation data available for learning. Second, these newkinds of information—“deep” annotations—are utilized in amachine-learning context. In particular, various embodiments utilize theconcept of oracle-assisted learning: learning in a setting wherepowerful, but opaque, black-box systems—in this case, humans—areavailable for interrogation. While in some instances it may remaininfeasible to dramatically increase the amount of data available fortraining of a given system, measuring the example-by-example difficultyand pattern of errors of the more powerful system may provide importantinformation for regularizing the solution of the system at hand.

Elements of an exemplary system 100 in accordance with an embodiment ofthe present invention are summarized in FIG. 1A, with three main areasof operation. As shown, an element 105 corresponds to crowd-sourcedvisual psychophysics, in which, e.g., human participants perform tasksvia a webform hosted at a public website. For example, a participant maybe shown (1) an image 110 of a person and asked that person's gender,(2) an image 115 of a partially obscured image (shown in FIG. 1A as aperson's face obscured by branches) and asked if a face is present,and/or (3) an image 120 of all or a portion of a person's face and askedto select an emotion that corresponds to the image, e.g., jealous,panicked, arrogant, or hateful. The results of element 105 are utilizedto inform an element 125, which corresponds to human-informed machinelearning training. As shown, in element 125, the outputs of element 105may be utilized to model patterns of error, which are then utilized toinform and constrain a machine vision system. Finally, in an element130, the machine vision system is utilized to solve any of a variety ofvisual-recognition problems. For example, in an application 135, thesystem may detect faces in images even in the presence of such obscuringfeatures as occlusion of the face or blur or noise in the image. In anapplication 140, the system may recognize and assign visual attributesto images. As shown, the system may recognize that an image contains aman and that the man is smiling, has a beard, and is wearing a hat.Finally, in an application 145, the system may be utilized to recognizevarious other objects in images; as shown, the system may identifyobjects such as a tree, a car, a bus, a building, and a street in agiven image.

As shown in FIG. 1B, in preferred embodiments of the invention,classification systems (or “classifiers”) are implemented on a server150 that includes a computer processor 155 and utilizes various programmodules. Program modules may include or consist essentially ofcomputer-executable instructions that are executed by a conventionalcomputer. Generally, program modules include routines, programs,objects, components, data structures, etc. that performs particulartasks or implement particular abstract data types. In accordance withvarious embodiments of the invention, data is classified and identifiedby initially providing training objects to multiple human annotators 160over a computer network 165. As used herein, a “computer network” is anywired and/or wireless configuration of intercommunicating computationalnodes, including, without limitation, computers, switches, routers,personal wireless devices, etc., and including local area networks, widearea networks, and telecommunication and public telephone networks. Thetraining objects may include or consist essentially of, for example,data representative of pictures, graphics, video and/or audio files,text or handwriting samples, etc. The subject matter of each trainingobject has certain features that may be classified by at least some ofthe human annotators 160. For example, the training objects may bedigital images containing, e.g., human faces for classification. Thetraining objects may be displayed to the human annotators by aconventional display module 167 via training devices 170; for example,the training devices 170 may be computers, cellular phones, or othermobile devices having audio, graphics, and/or video displays. Thedisplay module 167 is executable by the server 150 and/or its computerprocessor 155. As used herein, the term “display” means rendering thedigitally encoded subject matter of a training object so as to beperceivable by a user via any mode of presentation suitable to theparticular type of training object, including audio presentation via aspeaker or headphones and visual presentation via a monitor or otherdevice. The training devices 170 receive the training objects and/ortransmit classification data to the central server 150 via acommunication interface such as e-mail, SMS text message, or pages (or“webforms”) on hosted websites.

Multiple human annotators 160 annotate at least some of the trainingobjects via annotation interfaces such as touchscreens, touchpads,computer mice or other pointing devices, or keyboards, associated withthe training devices 170. During and/or after the annotation,psychometric data is also acquired that characterizes the annotation ofthe training objects by the annotators 160. For example, as describedherein in detail, such psychometric data may include or consistessentially of response times for classifying one or more features, theaccuracy of feature classification, and/or the presentation time (i.e.,the amount of time presented to each annotator) of one or more trainingobjects. The classification and psychometric data may be stored in adatabase 175 of training objects accessible by the server 150.

After acquisition of the classification and psychometric data, ahuman-weighted loss function is computationally derived based at leastin part on the psychometric data by, e.g., a penalization module 180executable by the server 150 and/or its computer processor 155. The lossfunction includes penalties for misclassification (of later presentedquery objects, as described below), and the magnitude of the penaltiesincreases with increasing deviation from the classification datareceived from the human annotators.

Once the human-weighted loss function is determined, one or more “queryobjects” may be received by the system for classification based on theclassification data and the loss function by, e.g., a classificationmodule 185 executable by the server 150 and/or its computer processor155. The query objects are new objects to be classified by the systemabsent direct human classification, and may be the same types of objectsutilized as training objects but having features that are different intype, placement, number, etc. The query objects may be received by aclassification device 190 such as a communication interface to a serveror database of objects, a camera or other image- and/or video-capturedevice, a microphone or other audio-capture device, etc.

Those skilled in the art will appreciate that embodiments of theinvention may be practiced with various computer system configurations,including multiprocessor systems, microprocessor-based or programmableconsumer electronics, minicomputers, mainframe computers, and the like.Embodiments of the invention may also be practiced in distributedcomputing environments where tasks are performed by remote processingdevices that are linked through a communications network. In adistributed computing environment, program modules may be located inboth local and remote computer-storage media including memory storagedevices.

Thus, systems in accordance with embodiments of the present inventionmay include or consist essentially of a general-purpose computing devicein the form of a computer including a processing unit (or “computerprocessor”), a system memory, and a system bus that couples varioussystem components including the system memory to the processing unit.Computers typically include a variety of computer-readable media thatcan form part of the system memory and be read by the processing unit.By way of example, and not limitation, computer readable media mayinclude computer storage media and/or communication media. The systemmemory may include computer storage media in the form of volatile and/ornonvolatile memory such as read only memory (ROM) and random accessmemory (RAM). A basic input/output system (BIOS), containing the basicroutines that help to transfer information between elements, such asduring start-up, is typically stored in ROM. RAM typically contains dataand/or program modules that are immediately accessible to and/orpresently being operated on by processing unit. The data or programmodules may include an operating system, application programs, otherprogram modules, and program data. The operating system may be orinclude a variety of operating systems such as Microsoft WINDOWSoperating system, the Unix operating system, the Linux operating system,the Xenix operating system, the IBM AIX operating system, the HewlettPackard UX operating system, the Novell NETWARE operating system, theSun Microsystems SOLARIS operating system, the OS/2 operating system,the BeOS operating system, the MACINTOSH operating system, the APACHEoperating system, an OPENSTEP operating system or another operatingsystem of platform.

Any suitable programming language may be used to implement without undueexperimentation the functions described above. Illustratively, theprogramming language used may include assembly language, Ada, APL,Basic, C, C++, C*, COBOL, dBase, Forth, FORTRAN, Java, Modula-2, Pascal,Prolog, Python, REXX, and/or JavaScript for example. Further, it is notnecessary that a single type of instruction or programming language beutilized in conjunction with the operation of systems and techniques ofthe invention. Rather, any number of different programming languages maybe utilized as is necessary or desirable.

The computing environment may also include other removable/nonremovable,volatile/nonvolatile computer storage media. For example, a hard diskdrive may read or write to nonremovable, nonvolatile magnetic media. Amagnetic disk drive may read from or writes to a removable, nonvolatilemagnetic disk, and an optical disk drive may read from or write to aremovable, nonvolatile optical disk such as a CD-ROM or other opticalmedia. Other removable/nonremovable, volatile/nonvolatile computerstorage media that can be used in the exemplary operating environmentinclude, but are not limited to, magnetic tape cassettes, flash memorycards, digital versatile disks, digital video tape, solid state RAM,solid state ROM, and the like. The storage media are typically connectedto the system bus through a removable or nonremovable memory interface.

The processing unit that executes commands and instructions may be ageneral-purpose computer processor, but may utilize any of a widevariety of other technologies including special-purpose hardware, amicrocomputer, mini-computer, mainframe computer, programmedmicro-processor, micro-controller, peripheral integrated circuitelement, a CSIC (Customer Specific Integrated Circuit), ASIC(Application Specific Integrated Circuit), a logic circuit, a digitalsignal processor, a programmable logic device such as an FPGA (FieldProgrammable Gate Array), PLD (Programmable Logic Device), PLA(Programmable Logic Array), RFID processor, smart chip, or any otherdevice or arrangement of devices that is capable of implementing thesteps of the processes of embodiments of the invention.

In accordance with various embodiments of the present invention,psychometric testing is organized around two primary measures: accuracyand response time. Each of these measures may be used on an individualimage, pooled for all subjects, or pooled for particular subjects. Thus,accuracy and reaction time may be measured on a per-image, per-subject,or per-condition level. When measured on a per-image level, meanaccuracy provides a measure of how difficult a given trial is over thesubject population. However, more refined information may often beobtained by performance per subject. Some “easy” images will beidentified by all subjects, no matter what their ability level, whereasothers will only be identified by highly skilled observers. Thisinformation may also be used effectively to target specific performancegoals for systems in accordance with embodiments of the presentinvention. FIG. 2 provides an example of the types of per-imageinformation that may be extracted using item-response testing and thenintegrated into the system. FIG. 2 depicts three item response functionsfor images presented on a crowd-sourced data-acquisition platform (or“classifier,” e.g., a website hosted by the server 150 connected to thecomputer network 165). The x-axis represents a latent measure of abilityon an underlying perceptual task (such as face or attribute detection).An easy curve 200 represents an image that differentiates subjects witha low latent ability to perform the detection task. A hard curve 210represents an image that differentiates subjects with a low latentability to perform the detection task. Each of the curves 200, 210 has asteep slope; this indicates that these images are very effective atdetermining whether a subject's overall performance is likely to be tothe left (better) or right (worse) compared to that curve on the x-axis.A problematic curve 220 has a very shallow slope; this means itdiscriminates between subjects poorly, as performance classifying theimage represented by curve 220 does not vary widely for subjects alongthe ability scale.

This process of determining the accuracy curves per image so as todesign more efficient tests is also referred to herein as item responsetesting (IRT). It is not the only method of extracting per-imagepsychometric data, but it is an exemplary embodiment of the type ofsecond-order statistical information that may be collected and appliedto the training of computer-vision systems in accordance withembodiments of the invention. One benefit of IRT and related approachesis that—because the x-axis is a transformed latent variable, as opposedto a parameterized manipulation—it is typically possible to determinedifficulty rankings for natural images that have not been parameterized;the transformation acts to pool information on item difficulty acrossall subjects.

Accuracy may be recorded in multiple ways; most commonly a response is abinary keypress or button click recorded on a per-trial basis. Whenmeasured per-subject, on many types of tests mean accuracy is comparedto a non-zero chance accuracy (so, for instance, on a two-alternativeforced choice (“2-AFC”) task, chance performance would be 50%). Whenmeasured on a per-stimulus level, mean accuracy provides a measure ofhow difficult a given trial is over the subject population. Such datamay be analyzed even more finely; for example, various embodimentsutilize mean accuracy per presentation time, presentation location, percondition, condition by subject, and so on. In these latter cases thefunction may be graphed as, and fit to, a psychometric function ofaccuracy per parameter, depending on the specific needs for a particulardeeply annotated training set.

Response time may be characterized as the sum of reaction time andmotion time for making a response. Preferred embodiments of the presentinvention utilize data sets in which the motion time is substantiallyconstant. Web-based response time may include another component—thetiming lag incurred by a browser (e.g., the browser's Javascriptexecution environment). However, in embodiments of the inventionresponse time (RT) may be analyzed in much the same way as accuracy:per-subject, per-stimulus, per-condition, and all the variations inbetween. Response time may provide useful additional information, as itmay serve as a difficulty metric on tasks (such as many face-perceptiontasks) where human subjects are basically perfect across the board. Evenif the mean accuracy of a given stimulus is 100%, it may have a longermean response time than another 100% accurate stimulus, indicatingrelatively greater difficulty. FIG. 3 depicts a graphical representationof a process 300 of recording psychophysical data from subjects inaccordance with embodiments of the invention. Specifically, FIG. 3 showsan example of a crowd-sourced data-acquisition work flow for anexperiment that involves the sequential presentation of some number n oftrials (featuring one or more images) to m visitors to a hosted website,where for each image that visitor has to make a classification decision;the displayed example is for a test that uses a key-press response, butthe basic design also applies to tests that use mouse clicks or othermethods of capturing subject responses. In a step 310, human annotators160 utilize training devices 170 to communicate with the server 150(e.g., via display module 167, which displays the hosted website) overthe computer network 165. In a step 320, n images (i.e., the trainingobjects) are presented to a number m of the annotators 160. In a step330, the annotators 160 utilize the training devices 170 to classify thetraining objects via, e.g., key-presses and/or mouse-button clicks. Thepsychometric data acquired during step 330, e.g., accuracy of imagecharacterization, response time, presentation time, etc., is recorded inthe database 175 in a step 340. The psychometric data may be displayedto analyze, e.g., response time per annotator in a graph 350, apsychometric function 360 related to the accuracy of the annotation percondition, and/or an item response curve 370 for each of the trainingobjects (i.e., the presented images).

The term “psychometric function,” as mentioned above, herein refers to afunction that relates accuracy (on either an individual subject orpooled-subject level) to a given parameter. There are many functionsthat fit this characteristic shape, but in various embodiments the mostimportant characteristics that the function describes are asub-threshold region where the participant performs at chance, a risingregion where performance improves rapidly, and a final roll-off of slopeas the subject reaches asymptotic (usually 100%) performance. Implicitin the shape of the psychometric curve is a sensory threshold, where thesubject is able to correctly perform the task some pre-selectedpercentage of the time. (The human performance curve depicted in FIG. 6Bshows one characteristic shape of a psychometric curve.)

As mentioned above, while in some instances it may remain infeasible todramatically increase the amount of data available for training of agiven system, measuring the example-by-example difficulty and pattern oferrors of more powerful systems may provide important information forregularizing the solution of the system at hand. With this in mind, FIG.4B depicts an overview of a learning approach in accordance with variousembodiments of the present invention, and FIG. 5 illustrates an exampleof prediction after learning in accordance with various embodiments.FIG. 4B depicts an embodiment of the present invention compared totraditional supervised learning, which is depicted in FIG. 4A. Previousattempts at incorporating humans into the machine learning process at amore detailed level have typically focused on a single human for thelabeling of difficult or ambiguous training data for continualimprovement of a class model. For example, in FIG. 4A, in a step 400, ahuman annotator assigns labels to a set of training objects, in a step405, the labeled data is utilized to train a machine classifier, and ina step 410, the classifier makes predictions in response to queriesprovided to the classifier.

In contrast, in embodiments of the invention, sets of queries are posedto a crowd, which may be composed of, e.g., citizen scientists (i.e.,annotators 160) connected via the computer network 165 to the server 150that collects, processes, and analyzes the data with the set ofprocessing modules detailed above. The crowd represents a powerful, butopaque black box implementation that is available for interrogation. Inthe framework of psychophysical experiments, patterns of error may bemodeled that may be translated to human-weighted loss functions forclassification, as well as penalties for margins that are not consistentwith human data during training. Thus, in a step 415, the annotators 160participate in psychophysical experiments involving the training objectsto produce psychometric data (e.g., as described above with respect toFIG. 3), and in a step 420, the data is utilized to model the pattern ofthe performance of the human annotators 160, resulting in ahuman-weighted loss function 425 computationally derived by thepenalization module 180. The loss function includes penalties formisclassification of later presented query objects, as graphicallyillustrated in graph 430, and the magnitude of the penalties increaseswith increasing deviation from the classification data received from thehuman annotators 160. Finally, in a step 435, the classification module185 is utilized to make predictions (based on various query objects)that are more consistent with the decisions of the human annotators 160.

FIG. 5 depicts an example embodiment of the invention using theViola-Jones detector as a basis for face detection. Classifiersconditioned by humans may be used as an augmentation to existingapproaches. The Viola-Jones detector is the most common algorithm usedfor the task of face detection because of its processing speed, forwhich it trades accuracy. By using a deeply annotated classifier,candidate face windows may be evaluated to a higher degree of accuracythan is possible with just the base Viola-Jones scoring approach. Asdepicted in FIG. 5, in an embodiment of the present invention theViola-Jones approach utilizes Haar-like basis functions 500, computesintegral images 510 for increased speed, and rapidly scans for candidatewindows 520 at different scales. In a step 530, the Viola-Jones scoringprocess is augmented by utilizing the candidate face windows with thedeeply annotated classification system described above (e.g., toclassification module 185). In a step 540, false positive facedetections are filtered out utilizing the system in accordance withembodiments of the invention, and in a step 550, the best window isselected from a set of possibly valid face detections.

Active learning is one approach to incorporate humans into themachine-learning process at training time. As an alternative,embodiments of the present invention treat the oracle as a large set ofparticipants using crowd-sourcing (i.e., data acquisition from a largepopulation of human annotators 160), e.g., a crowd-sourced citizenscience website, as opposed to relying on just a single human annotator.The query process may employ the psychophysical experiments and datadescribed above. Instead of an algorithm querying the oracle in specificcases where difficult or ambiguous data is encountered, the queries inaccordance with embodiments of the invention typically occur prior totraining and classification, and encompass numerous visual examples.Once the human responses have been analyzed to quantify dominantpatterns of error, the models may be translated into human-weighted lossfunctions for maximum-margin kernel machines and penalties for marginsnot consistent with human data.

Generally, in any solution to a recognition problem, there is somenotion of risk involved that indicates the penalties incurred if aprediction is incorrect. Statistical learning may seek to find arecognition function f that minimizes the ideal risk RI:

${\underset{f}{argmin}\left\{ {{R_{I}(f)}:={\int_{{\mathbb{R}}^{d} \times {\mathbb{N}}}{{\phi\left( {x,y,{f(x)}} \right)}{P\left( {x,y} \right)}}}} \right\}}\ $R_(I) is composed of two terms, the joint distribution of data andlabels P(x, y), and the loss function φ(x, y, f(x)), which assigns thecost of misclassification. One component of a human-regularizedmaximum-margin kernel machine is the addressing of the issue of the lossfunction. A prediction during training may be calculated as the outputof the classifier for a particular training sample multiplied by itslabel: z=yf(x). In accordance with various embodiments of the invention,a loss function that applies a linearly increasing cost formisclassifications (one-sided error) is desirable. This is embodied bythe hinge loss function, which is defined as:φ(z)=max(0,1−z).

However, the non-linear nature of psychometric curves for visualrecognition tasks suggests a model that is much different than linearloss growth when z<1. Therefore, embodiments of the invention assume analternative model where wt(x) is a weighting function conditioned on apsychometric model for a recognition task t. The weights are determinedby the classification data collected from the annotators 160, includingthe individual and combined statistics of accuracy, response time andpresentation time, over a sampling of images across varying degrees ofdifficulty. Human-weighted loss is thus defined as:φ(z)=max(0,w _(t)(z)).

In this formulation, there is more risk associated with misclassifyingthe deeply annotated training samples, which forces more consistencywith the human model for the training data.

Various embodiments of the present invention also directly manipulatemargins. Considering a typical support vector machine (SVM) framework, amargin between a decision boundary (hyperplane) and closest data pointis defined during training to maximize the separation between classes.As a simple illustration of an embodiment of the invention, consider themost basic linear case, where the margin space is expressed asγ=y_(i)(w·x_(i)−b)≧1. w is the normal vector to the decision hyperplane,xi is a real valued vector, and b is an intercept term that helpscontrol the offset of the decision hyperplane from the origin along w.The goal of the SVM optimization process, in accordance with variousembodiments, is to maximize parameter

$\frac{2}{w}$by minimizing ∥w∥. Surprisingly, points near decision boundaries tend tobe the most interesting for many recognition problems—knowledge of howhumans treat the points in this space may potentially improve marginslearned during training. Leveraging gathered psychometric data,embodiments of the invention add another penalty for margins notconsistent with human data to the training formulation of commonmaximum-margin classifiers. During training, the distance betweenpotential margins γm produced by the algorithm in the optimization phaseand a human model γh for the problem may be calculated via δ=d(γm, γh).The resulting penalty δ may be assigned as an additional weight term:∥w+δ∥. The larger the distance between a potential margin and theoracle-defined model, the larger ∥w+δ∥ grows, resulting in a narrowmargin that diverges from human expectations. Learning strategies inaccordance with various embodiments of the invention incorporate thisnew margin penalty plus the loss function described above.

In an implementation to serve as a second-stage filter for objectdetection, embodiments of the invention may train and test on featurevectors computed in a dense grid from an image. Since an exhaustive scanof an image using a sliding window and SVM at multiple scales is oftenprohibitively expensive computationally, the detection algorithm mayleverage a boosted cascade of features as a first stage. By relaxing theneighborhood scoring constraints of such detectors and increasing thenumber of scales searched by the algorithm, a large number of candidateface patches may be collected. A deeply annotated linear SVM may then beused as a second-stage filter. Patches that are positively identified bythe SVM may be grouped into neighborhoods, filtered for redundancy, andscored to produce a set of final detection predictions.

Face detection is a common problem faced by machine-vision systems. Thisis because the correct localization of faces and within-face landmarksare often necessary preconditions to any higher-order classification (ofidentity or attributes, for example). Due perhaps in part to thisnecessity, humans have an exceptionally well developed face-detectionability, one which has not yet been matched by algorithmic performancedespite the high level of attention paid.

One of the issues for efforts to compare human and computer performanceon face-recognition tasks is the difficulty of quantifying the humanlearning rate for a given face; while human observers may have seen themost familiar faces hundreds of thousands of independent times, or more,the learning gradient with unfamiliar or partially familiar faces is noteasily accessible to empirical investigation. With face detection, bycontrast, all human observers are essentially maximally trained experts;this ability is present or begins developing shortly after birth, so thesize of the training set may be considered essentially unbounded andperformance essentially asymptotic.

Frontal face detection has been an early commercial success in computervision; the ubiquitous Viola-Jones algorithm of 2001 is both efficientenough to run on a microcontroller and relatively high-performing.Google's StreetView product's ongoing viability is predicated on theefficacy of their face detection and blurring algorithms for privacy.Indeed, the efficacy of Google's algorithm could be all that standsbetween the company and significant legal liability in many countries.Finally, Facebook is moving quickly into the face detection andrecognition space, buying the industry-leading technology of face.com.

In addition to commercial activity, the amount of existing research onface detection is substantial. This provides a tremendous boon forhigh-throughput comparison methodologies, because the landscape ofapproaches that have been tried is both vast and heterogeneous; whilecertain technologies have proven themselves commercial winners, thereare any number of different approaches that have had more-or-less broadsuccess at face detection within the commonly used reference datasets,including LFW, AFW, Multi-PIE, and others. Certain algorithms mayaddress the inherent difficulty of detecting unconstrained faces bylearning local and global features, often updating models as newinformation is encountered. In this mode, component-based frameworks,online domain adaptation, and mixtures of tress with a shared pool ofparts may be potential solutions. However, even acceptable tolerance tothe common effect of occlusion has yet to be achieved by existingalgorithms.

Face detection is also a promising task for the creation ofdeeply-annotated datasets in accordance with embodiments of the presentinvention because the ecological task includes many naturaldegradations; face detection in human observers happens in conditionsthat may include many distinct types of occlusion, among them naturalobstructions (trees and other landscape features), man-made obstructions(e.g., cars, fences, windows and other features of the man-madeenvironment), weather conditions (snow, rain, fog) and other people (as,for instance, in crowded environments). All of these occlusion scenariosare relatively common, and all of them evidently present challengeswhere face-detection system would optimally be expected to perform, at aminimum, at the level of a human observer.

Human performance generally still greatly exceeds machine performance.On a noise-added task, the best available commercial algorithms (asimplemented in Google's Picasa software and in face.com's web-accessibleAPI) perform well below the level achieved by human observers even withrelatively brief presentation times, as shown in FIGS. 6A and 6B. Asshown in FIGS. 6A and 6B, embodiments of the invention reduce thedivergence between human and machine performance. FIG. 6A depictsdifferent levels of face coherence constructed using random phasescrambling in images 600, 610, 620 having face coherences of 0.8, 0.6,and 0.4, respectively. FIG. 6B compares performance of human subjectsand computer algorithms on the ability to detect face stimuli. The line630 indicates human performance, while the lines 640, 650, 660, and 670represent the performance of various computer algorithms.

The divergence between human and algorithmic performance is even greaterwhen the generated images are changed to represent a solid-occluderscenario intended to have greater ecological validity, as shown in FIGS.7A and 7B. The relative strengths and weaknesses of human observersdiffer greatly from those of the most successful computer algorithms,and humans are able to solve reasonably natural face detection problemsthat conventional computer algorithms cannot closely approach. Even moreso than in the case of noise above, embodiments of the inventiondirectly address deficiencies in algorithmic performance for thedetection of occluded faces. FIG. 7A depicts three examples 700, 710,720 of occluded faces used as stimuli, and FIG. 7B depicts a comparisonof human and computer algorithm performance. The line 730 shows humanperformance for faces occluded with a large number of relatively thinoccluders, transposed and normalized for a 100% scale. The other threelines 740, 750, and 760 represent the performance of computeralgorithms.

Embodiments of the present invention incorporate a substantial pool ofdetection data (e.g., more than ten thousand subjects with one hundredtrials per subject across all conditions, tested with approximately7,500 face images) for occluded faces in very large, noisy,perpendicularly obstructed, and Simoncelli textured configurations.These scenarios represent an important set of difficult circumstancesfor detection, enabling the location of more faces through a betterunderstanding of difficult examples during classifier training.

While embodiments of the present invention are more able than othermachine learning approaches to make use of manipulated or “lab” stimuli,they also may accommodate natural, uncontrolled face images that presentdifficulty to a detector, especially those that are not in a frontalpose. Faces, such as that in image 800, are easily recognized by humansubjects in a massive range of poses because of the invariant nature ofobject recognition in the brain. To address this, embodiments of theinvention deploy a test using uncontrolled natural images 810 whereannotators are brought off ceiling by using very brief (e.g.,approximately 50 ms) presentation times and masking withamplitude-matched phase-scrambled noise, as shown in FIG. 8. Forexample, annotators may be asked to press a specific key if they observea face in an image 810 and another key if they do not observe a face inthe image. At the end of a series of such queries, end-of-teststatistics 820 may be displayed to the annotators 160 via display module167. A complete face-detection process leveraging the Viola-Jonesdetector as a basis is depicted in FIG. 5. FIGS. 8A and 8B depict teststhat may be utilized in embodiments of the present invention, includinga face-detection task with real-world face images that uses rapidpresentation and masking to bring subjects off ceiling, and an abstractattribute identification test to increase understanding of the limits ofvisual label assignment. This data may be used to inform machinelearning in accordance with embodiments of the invention.

Visual attributes are a powerful representation for a variety of visiontasks including recognition, classification, and image description. Inessence, attributes are semantically meaningful labels that are used byhumans to describe parts of a scene. For instance, considering the humanface, it may be said that a woman is “beautiful,” a man has “blackhair,” or an emotional state may be described as “happy” or “sad.” Theattribute model for recognition is compelling at a psychological level,but has been inhibited computationally by the definition of therepresentation and how it is learned. These problems have been partiallyaddressed by advances in machine learning, where many thousands oftraining images from the Internet may be harnessed to create attributeclassifiers that generalize reasonably well. Attributes may beconsidered individual “building blocks” for higher-level recognitiontasks, with their own complexities and challenges regarding features andtraining. Thus, rather than initially tackling the more complex problemof face identity, embodiments of the invention consider attributes as adistinct problem. Existing algorithms for attributes have not beeninformed by human behavior by any significant measure, makingembodiments of the invention a novel alternative.

Current attribute models only hint at the utility of human models forimproved supervised learning. By quantifying human patterns of errorover large sets of attributes and large populations, embodiments of theinvention produce well-informed machine learning classifiers that aremore effective for this task. Another unique aspect of embodiments ofthe present invention is the level of abstraction that may be modelled.Prior work in automatic classification has focused on prevalent facialfeatures and easily assignable context, but there is no work to date onhighly abstract attributes, even when those attributes may be reliablyrated by human subjects. To this end, embodiments of the inventionwebsite capture attributes such as “dominance” and “trustworthiness,” asshown in FIG. 8B. In accordance with various embodiments, a behavioralmeasure asks annotators 160 to give pairwise-ratings for pairs 830, 840of training objects, e.g., which of the two images in each pair is “moretrustworthy” or “more dominant.” Pairs 830, 840 are exemplary, andannotators 160 will typically be presented with many more pairs duringtraining, e.g., one hundred sets of faces from within a larger set offace data. These pairwise comparisons are used to create an estimatedranking for each face in the dataset, weighted by annotators' meanresponse time. These attributes are important in a psychologicalcontext, where quick judgments of power relationships and personalitycharacteristics may be vital to social interaction, as well as in acomputer-vision context, where—in addition to their possible usefulnessin applications such as threat detection—the need to understand thelimits of attribute quantification has not been adequately addressed.

With the prevalence of machine-learning-based algorithms for generalobject recognition, embodiments of the invention may apply to numerousexisting approaches as a second-stage filter to process results, or as adirect replacement for supervised classification components. Individualclassification systems for any visual category may be trained withdeeply annotated images, by following the learning procedure withhuman-weighted loss and margins that are penalized for beinginconsistent with those established by human annotators. Deeplyannotated data may be obtained by adding new tasks to the crowd-sourceddata-collection penalization module (via display of, e.g., a citizenscience website), facilitating recognition of broad categories ofobjects including faces for identity purposes.

EXAMPLE

To validate embodiments of the invention, three different aspects wereevaluated: (1) the impact of replacing the hinge loss function with thehuman-weighted loss function for SVM image classification, (2) theperformance of a deeply annotated detector on a standard benchmark dataset, and (3) the impact of replacing hinge loss with human-weighted lossin the face-detection scenario. Data from the Face Detection Data Setand Benchmark (FDDB) set, the most current benchmark for unconstrainedface detection, was utilized. In this example, FDDB consisted of 2,845images that contain 5,171 annotated faces, split across 10 differentfolds for cross-validation-style testing. The set contains a wide rangeof challenges including occlusions, difficult poses, and low resolutionand out-of-focus faces, making it quite suitable for this evaluation.

To assess the impact of the choice of loss function during training, aseries of classification experiments using data sampled from the FDDBset was completed. For each face in each image, a corresponding negativepatch by randomly sampling from the nonface image regions was selected.This yielded ten folds of data with balanced positive and negativeexamples to be used for training and testing. A classifier for each ofthe ten folds was then trained using 200 images drawn directly from theFDDB data and 100 images from a deeply annotated set incorporatingSimoncelli textures using recorded human accuracy as the weights in thehuman-weighted loss function, and tested on the remaining nine, for atotal of 90 classification tests. The results are shown in FIG. 9. FIG.9 depicts linear SVM face-classification results for an embodiment ofthe invention utilizing a human-weighted loss function (curve 900)compared to use of the common hinge loss function (curve 910). Eachpoint represents the average accuracy of nine classification foldsproduced by a classifier trained on one fold of FDDB (leave-one-outcross validation). The error bars represent standard error. Allclassifiers were trained with 200 images from FDDB and 100 images fromthe TestMyBrain Simoncelli textures set, and share a C parameter of1×10−9. The testing data was composed of all ground-truth faces plus anequally sized set of randomly sampled negative patches. In every case,the human-weighted loss function exhibited improved accuracy, with anaverage improvement of 4.86% over hinge loss.

To assess the performance of a face detector in accordance withembodiments of the invention, the FDDB continuous score protocol withoutside training data was considered. The results are shown in FIG. 10.Compared to the existing peer-reviewed work reporting results on thistest, embodiments of the present invention have the highest level ofaccuracy. Specifically, line 1000 corresponds to the true positive rateas a function of the number of false positives in accordance with anembodiment of the present invention, while lines 1010, 1020, 1030, and1040 correspond to results obtained via conventional techniques. Asshown, various embodiments of the invention utilize the baselineperformance for the Viola-Jones detector as a basis for the classifier.By adding an additional face-filtering stage that incorporates deeplyannotated classifiers, embodiments of the invention improve the accuracyof Viola-Jones by a large margin. Thus, compared to the conventionalresults on this test, embodiments of the invention are more accurate. Toensure that these gains were not an artifact of good features and thebase SVM approach, an additional experiment was conducted on the samescore protocol with the detector making use of either human-weightedloss or hinge loss. FIG. 11 depicts face-detection results for the FDDBcontinuous score protocol showing a detection approach implemented withhuman-weighted loss in accordance with embodiments of the invention(curve 1100), as well as for hinge loss for comparison (curve 1110).Like the classification scenario of FIG. 9, FIG. 11 shows a noticeableincrease in accuracy for face detection by using human-weighted loss.

A selection of qualitative results are provided in FIG. 12, whichindicates that embodiments of the present invention not only providebetter recall performance, but also greater tolerance for pose,detection when a face fills the frame, and the detection of stronglyoccluded faces. FIG. 12 depicts a qualitative comparison between facedetection performed by an embodiment of the present invention (upper rowof images, with detected faces outlined with rectangles) and byViola-Jones as implemented in the OpenCV Library (lower row of images),one algorithm that may be utilized as the basis for the detectionpipeline in accordance with embodiments of the invention. The facedetection performed in accordance with the embodiment of the presentinvention not only provides better recall performance, but also greatertolerance for pose (as shown in the first two images from the left, inwhich the embodiment of the present invention detects more faces despitedifferent pose angles), detection when a face fills the frame (as shownin the middle image, in which the embodiment of the present inventionsuccessfully detects the face), and the detection of strongly occludedfaces (as shown in the first two images from the right, in whichoccluded faces are successfully detected by the embodiment of thepresent invention).

Embodiments of the present invention may be directly applied to severalimportant domains where machine learning is found either as anaugmentation to an existing algorithm, or as a standalone solution. Asdescribed above, computer vision is a relevant area where the success ofembodiments of the invention has been demonstrated for the problem offace detection; embodiments of the invention are also applicable tovisual attributes and general object recognition. There are additionalproblem areas within computer vision that may also be addressed byembodiments of the invention, including scene classification, sceneunderstanding, optical character recognition, pose estimation, imageretrieval, target detection and tracking, image segmentation andautomated visual saliency. In the discipline of human biometrics,embodiments of the invention may be applied to face image pair matching,face verification, face identification, fingerprint verification,fingerprint identification, iris verification, iris identification, gaitrecognition, handwriting recognition, novel modality matching, andmulti-biometric fusion. Important application areas may also benefitfrom embodiments of the invention, including video surveillance,physical access and authentication, image and video forensics,photo-tagging for social media, computational photography,human-computer interaction (for ordinary users and the disabled), videoentertainment/games, robotics, image analysis for biomedicalapplications, intelligent content archiving, driverless/semiautonomousautomobiles, and machine vision for manufacturing.

Beyond vision, there are important problem areas in audio processing towhich embodiments of the invention may be applied, including voicerecognition, music analysis, audio retrieval, machine translation, andemotion recognition. As in vision, audio-based human biometrics such asspeaker verification and speaker identification may be addressed by theembodiments of the present invention. Specific application areas inaudio processing where embodiments of the invention apply include audiosurveillance, psychical access and authentication, entertainment,human-computer interaction (for ordinary users and the disabled), andintelligent content archiving.

Related to the processing of voice signals, the domain of textprocessing seeks to understand natural language in an intelligent waybased on typed or handwritten characters. Embodiments of the inventionapply to this area as well, including the specific problems of naturallanguage processing, machine translation, authorship attribution, topicmodeling, computer/formal language processing, and bioinformatics.Specific application areas in text processing where embodiments of theinvention apply include search engines, social media analysis (sentimentclassification, content summary, textual reuse identification),plagiarism detection, text-based forensics, network intrusion detection,and computer source code profiling.

Finally, the domain of financial analysis incorporates machine learninginto a number of problem areas, including market research, marketforecasts, and portfolio optimization. Embodiments of the presentinvention may be applied to all such areas.

It is noted that embodiments of the present invention are notconstrained to maximum-margin kernel machines. Embodiments of thepresent invention may be applied to any form of supervised learning,including neural networks, boosting, bagging, random forests, nearestneighbor algorithms, naive bays classifiers, density estimators, andother forms of statistical regression. Embodiments of the presentinvention may also be applied as part of a supervised component ofsemi-supervised or deep-learning algorithms.

In addition to the psychometric measures already mentioned, otherpsychometric measures which may be used to generate deeply annotatedtraining data within embodiments of the present invention includepsychophysical behavioral measures: item response slope, item responseoffset, mouse path, pairwise ranking, likert scale judgment, continuousscale judgment, continuous color report, and mouse deviation from targetlocation. All of these measures may be weighted by the latent subjectability measure theta.

In addition to behavioral measures, event-related potentials may be usedby annotating training data with the slope of the ERP response, theamplitude of the ERP response, the gross magnitude of the ERP responseand the divergence of the ERP response from mean, as well as the meandivergence of the ERP response from resting baseline. Relatedmagnetoencephalography measures may also be used, such as MEG signalamplitude, MEG signal slope and MEG signal Euclidean distance from themean, or MEG signal Euclidean distance from baseline. Magnetic resonanceimaging may be used to deep annotate training samples by means ofEuclidean distance of signal from mean signal or Kullback-Lieblerdivergence of signal from mean signal.

Eye-tracking hardware may be used to collect measures suitable for deepannotation such as saccade-to-target accuracy, saccade-to-targetlatency, number of saccade hops to target, and total number of saccades.

The stimuli that are presented to observers may be degraded by a numberof techniques other than the noise and occlusion previously mentioned.These include, for visual stimuli, l/f visual noise, white visual noise,pink visual noise, amplitude-matched phase scrambling, proceduralocclusion, second-order statistic-matched backgrounds, second-orderstatistic-matched blended noise, very brief presentation time, onetarget, multiple foil search displays, contrast lowering, brightnesslowering, false color, inversion, image scrambling, backward maskingwith noise, backward masking with images, rapid serial visualpresentation (RSVP), blur, lightening, specular highlighting, foil imagesuperimposition or composition, generated shadows, outline drawings,two-tone images, silhouettes and selective spatial frequency removal.

For auditory stimuli, the degradation techniques that may be appliedinclude additive white noise, additive pink noise, additive brown noise,overlapped audio, random segment silencing, tempo shifting, pitchshifting, selective frequency removal, phoneme removal, phonemereplacement, and audio scrambling by permutation of arbitrarily sizedsegments.

For text-based stimuli, the degradation techniques that may be appliedinclude selective word omission, word replacement, the use of difficultfonts, rapid presentation of stimuli, presentation of short snippets ofstimuli, and inverted stimuli.

The terms and expressions employed herein are used as terms andexpressions of description and not of limitation, and there is nointention, in the use of such terms and expressions, of excluding anyequivalents of the features shown and described or portions thereof. Inaddition, having described certain embodiments of the invention, it willbe apparent to those of ordinary skill in the art that other embodimentsincorporating the concepts disclosed herein may be used withoutdeparting from the spirit and scope of the invention. Accordingly, thedescribed embodiments are to be considered in all respects as onlyillustrative and not restrictive.

What is claimed is:
 1. A computer-implemented method for dataclassification and identification, the method comprising: providing,over a computer network, data corresponding to a plurality of trainingobjects to a plurality of training devices each associated with one of aplurality of human annotators, each of the training objects comprisingfeatures for classification; displaying the training objects on adisplay of each of the training devices; receiving, via communicationinterfaces of at least some of the training devices, classification datacomprising at least some of the training objects annotated, viaannotation interfaces of the training devices, by at least some of theannotators with classifications for features thereof; acquiringpsychometric data characterizing the annotation of the training objectsby the annotators; computationally deriving a human-weighted lossfunction based at least in part on the classification data and thepsychometric data, the loss function comprising penalties formisclassification, magnitudes of the penalties increasing withincreasing deviation from the classification data; receiving, by aclassification device, data corresponding to a query object differentfrom the plurality of training objects; and thereafter, computationallyclassifying, by a computer processor, at least one feature of the queryobject based at least in part on the human-weighted loss function. 2.The method of claim 1, further comprising displaying the classificationof the at least one feature of the query object.
 3. The method of claim1, wherein the magnitudes of the penalties for misclassification arebased at least in part on the psychometric data.
 4. The method of claim1, wherein computationally deriving the human-weighted loss functioncomprises: computationally classifying, by the computer processor, atleast some of the training objects based at least in part on an initialloss function, thereby generating training data; comparing the trainingdata to the classification data to identify, within the training data,features misclassified in comparison to the classification data;assigning the penalties for misclassification to the misclassifiedfeatures in the training data; and incorporating the penalties formisclassification within the initial loss function to generate thehuman-weighted loss function.
 5. The method of claim 4, wherein thepenalties for misclassification are assigned based at least in part onthe psychometric data.
 6. The method of claim 4, wherein the initialloss function comprises a hinge loss function.
 7. The method of claim 1,wherein the psychometric data comprises at least one of (i) responsetime for classifying one or more features, (ii) accuracy of featureclassification, or (iii) presentation time of one or more trainingobjects.
 8. The method of claim 1, wherein each of the training objectscomprises a digital image, and one or more features for classificationcomprise human faces.
 9. The method of claim 1, wherein at least one ofthe training devices comprises a computer or mobile computing device.10. The method of claim 1, wherein the query object comprises a digitalimage, and at least one said feature of the query object comprises ahuman face.
 11. A system for data classification and identification, thesystem comprising: a database of training objects, the databasecomprising a storage medium populated with stored computer recordsspecifying, for each of a plurality of training objects, (i)classification data comprising annotations received from a plurality ofhuman annotators, and (ii) psychometric data characterizing theannotation of the training object by the plurality of human annotators;a computer processor; a classification device for receiving queryobjects different from the training objects in the database; apenalization module, executable by the computer processor, for derivinga human-weighted loss function based at least in part on theclassification data in the database and the psychometric data of atleast some of the training objects in the database, the loss functioncomprising penalties for misclassification, magnitudes of the penaltiesincreasing with increasing deviation from the classification data; and aclassification module, executable by the computer processor, forclassifying features of query objects based at least in part on thehuman-weighted loss function.
 12. The system of claim 11, furthercomprising a display module, executable by the computer processor, fordisplaying training objects to the plurality of human annotators. 13.The system of claim 12, further comprising a plurality of trainingdevices, each associated with a human annotator, for displaying trainingobjects, each training device comprising a communication interface forreceiving training objects and transmitting classification data.
 14. Thesystem of claim 13, wherein at least one of the training devicescomprises a computer or mobile computing device.
 15. The system of claim11, wherein the psychometric data comprises at least one of (i) responsetime for classifying one or more features, (ii) accuracy of featureclassification, or (iii) presentation time of one or more trainingobjects.
 16. The system of claim 11, wherein each of the trainingobjects comprises a digital image, and one or more features forclassification comprise human faces.
 17. The system of claim 11, whereinat least one query object comprises a digital image, and at least onefeature of the query object comprises a human face.